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ABSTRACT 

In  the  field  of  military  Unmanned  Ground  Vehicles  (UGV),  military  units  are  adapting  their  concept  of  operations  to  focus 
on  their  mission  capabilities  within  populated  cities  and  towns.  These  types  of  operations  are  referred  to  as  MOUT  (Military 
Operations  on  Urban  Terrain).  As  more  Soldiers  seek  to  incorporate  technology  to  enhance  their  mission  capabilities,  there 
then  becomes  a  need  for  UGV  systems  to  encompass  an  ability  to  autonomously  navigate  through  urban  terrains. 
Autonomous  systems  have  the  potential  to  increase  Soldier  safety  by  mitigating  the  risk  of  unnecessary  enemy  exposure 
during  routine  urban  reconnaissance. 

This  paper  presents  the  development  and  methodology  that  the  military  has  sought  to  increase  mission  capabilities  by 
incorporating  autonomy  into  manned/unmanned  ground  vehicles.  The  presented  solution  that  has  been  developed  through 
the  Safe  Operations  of  Unmanned  systems  for  Reconnaissance  in  Complex  Environments  (SOURCE)  Army  Technology 
Objective  (ATO)  has  the  ability  and  has  been  tested  to  safely  navigate  through  complex  urban  environments.  This  paper  will 
also  focus  on  the  challenges  the  military  has  faced  to  develop  the  presented  autonomous  UGV. 
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1.  INTRODUCTION 

As  Warfare  evolves,  military  technology  must  also  evolve  to  support  and  aid  current  battlefield  requirements  and  threats. 
The  military  is  presently  adapting  its  concept  of  operations  to  take  advantage  of  the  relatively  new  technology  of  Unmanned 
Ground  Vehicles  (UGV’s),  with  a  focus  on  their  mission  capabilities  in  populated  cities  and  towns.  These  types  of 
operations  are  referred  to  as  MOUT  (Military  Operations  on  Urban  Terrain).  MOUT  missions  typically  involve  interaction 
with  civilian  populations,  where  it  can  be  difficult  to  determine  who  is  friendly,  hostile,  or  ambivalent.  Additionally,  much 
of  the  environment  is  man-made,  making  it  difficult  to  detect  modifications  by  the  enemy.  This  close  proximity  to 
potentially  hostile  forces  makes  operations,  such  as  military  vehicle  convoys  that  are  manually  driven  and  occupied  by 
Soldiers,  increasingly  dangerous.  As  more  Soldiers  request  technology,  such  as  UGV’s,  to  enhance  their  mission 
capabilities,  researchers  and  developers  are  pushing  to  increase  the  performance  of  military  vehicles  that  can  autonomously 
navigate  through  urban  terrains.  Autonomous  systems  have  the  potential  to  increase  Soldier  safety  by  mitigating  the  risk  of 
unnecessary  enemy  exposure  during  routine  urban  missions,  such  as  logistics  and  reconnaissance,  by  removing  the  Soldier 
from  direct  contact  with  potential  enemies. 

For  this  reason,  this  paper  will  present  the  development  and  methodology  that  the  Army  has  sought  to  increase  mission 
capabilities  through  the  incorporation  of  autonomy  to  exchange  manned  ground  vehicles  for  UGVs.  The  presented  solution 
that  has  been  developed  through  the  Safe  Operations  of  Unmanned  systems  for  Reconnaissance  in  Complex  Environments 
(SOURCE)  Army  Technology  Objective  (ATO)  has  the  ability  and  has  been  tested  to  safely  navigate  through  complex  urban 
environments.  This  paper  will  also  focus  on  the  challenges  the  military  has  faced  to  develop  the  presented  autonomous 
UGV. 

The  objective  of  the  SOURCE  ATO  is  to  advance  existing  UGV  technology  development  specifically  in  the  area  of  MOUT 
missions.  The  three  major  experiments  of  the  SOURCE  ATO  provide  opportunities  to  advance  the  development  and 
assessment  methodologies  for  UGV’s  in  more  complex  environments.  The  SOURCE  Enhanced  Experiment  is  the  second  of 
the  three  and  was  completed  in  November  2011. 

The  site  of  the  SOURCE  Enhanced  Experiment  site  was  the  MOUT  Facility  in  Camp  Lejeune,  North  Carolina.  The  MOUT 
site  is  an  urban  environment  including  paved  and  gravel  roads  lined  with  one,  two  and  three  story  buildings.  A  traffic  circle 
and  underpass  are  also  included  at  the  site  as  well  as  a  road  that  loops  the  outer  perimeter  of  the  site.  The  MOUT 
environment  is  complex  in  that  the  buildings  are  placed  closely  together,  and  the  outer  loop  road  is  densely  lined  with  trees. 
The  experiment  included  static  obstacles,  static  and  moving  pedestrians  (mannequins)  and  static  and  moving  vehicles.  The 
SOURCE  UGV  was  required  to  autonomously  operate  safely,  avoid  the  obstacles  and  obey  normal  rules  of  the  road,  during 
multiple  scenarios  within  this  environment.  The  development  and  methodology  described  in  this  paper  includes  information 
gained  from  the  SOURCE  Enhanced  Experiment. 
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Figure  1.  Enhanced  Experiment  MOUT  Course 

2.  COMPLEX  ENVIRONMENT 

Autonomous  platforms  require  more  robust  sensing  and  greater  intelligence  as  the  environment  becomes  more  complex. 
Urban  environments  often  have  very  complex  situations  occurring  in  very  constrained  spaces.  The  autonomous  platform  is 
required  to  not  only  navigate  itself  through  the  town  or  village,  avoiding  static  obstacles  and  structures,  but  also  to  obey  the 
rules  of  the  road,  including  the  local  unwritten  rules,  which  can  often  only  be  learned  by  experience.  The  vehicle’s 
perception  system  must  also  interpret  street  signs,  traffic  lights,  and  lane  markings,  as  well  as  anticipate  what  pedestrians 
and  other  vehicles  intend  to  do. 

Urban  obstacles  can  include  curbs,  speed  bumps,  potholes,  debris,  trash,  street/road  signs,  telephone  and  electric  poles,  fire 
hydrants,  bridges,  and  water.  The  UGV  needs  to  safety  navigate  around  these  obstacles,  without  causing  damage  to  itself  or 
to  them.  Urban  structures  are  special  cases  of  urban  obstacles,  since  they  are  more  likely  to  contain  people,  whether  friendly 
or  hostile.  Items  in  this  category  include  buildings,  bus  stops,  sidewalks,  and  markets. 

Additionally,  it  also  important  for  the  vehicle  to  respond  properly  to  objects  that  do  not  pose  a  threat,  but  this  becomes 
especially  challenging  when  dealing  with  humans  or  animals  that  may  change  trajectory,  suddenly  and  without  warning.  The 
vehicle  system  will  not  be  able  to  operate  effectively  if  it  does  not  put  some  bounds  on  the  expected  behavior  of  potential 
moving  objects,  including  people  and  animals.  In  urban  environments,  the  number  of  potential  vehicle  to  civilian  encounters 
increases  exponentially  as  the  urban  population  density  increases.  In  a  complex  urban  environment,  the  UGV  will  be 
required  to  detect,  identify,  and  avoid  people  (and  animals)  in  all  states  of  motion,  whether  crossing  the  street  at  a  crosswalk 
or  not,  walking,  running,  or  jogging  at  various  angles  with  respect  to  the  UGV’s  motion,  or  perhaps  just  standing,  sitting,  or 
lying  near,  or  even  on,  the  roadway.  Due  to  the  random  behavior  of  people  (or  animals),  predicting  their  motion  and 
planning  routes  around  them  can  be  very  difficult.  Even  people  have  difficulty  in  performing  this  task,  which  results  in 
many  traffic  accidents  and  fatalities  throughout  the  world.  However,  fair  or  not,  it  is  likely  that  autonomous  vehicles  will,  at 
least  initially,  be  held  to  a  higher  standard  of  performance  than  human  drivers. 

While  the  UGV  is  required  to  safely  navigate  itself  around  people,  obstacles  and  structures,  it  is  also  necessary  to  maintain 
the  rules  of  the  road  and  right-of-way  by  identifying  other  vehicles  operating  on  the  road,  waiting  at  intersections,  or  parked 
alongside  the  road.  The  rules  of  the  road  require  the  UGV  to  recognize  lane  markings,  street  signs,  and  traffic  lights  and  to 
react  accordingly.  However,  in  a  military  operation,  the  rules  of  the  road  and  who  has  right-of-way  can  change.  It  may  be 
necessary  for  the  autonomous  vehicle  to  push  its  way  through  traffic,  or  even  onto  a  sidewalk  or  off-road,  ignoring  the 
traditional  rules-of-the-road.  These  situations  may  require  more  human  intervention,  either  through  direct  teleoperation, 
semi-autonomous  control,  or  a  leader/follower  arrangement. 

If  these  were  not  complex  enough,  the  vehicle  must  also  perform  in  a  variety  of  weather  and  lighting  conditions,  which  can 
affect  the  UGV’s  sensor  inputs,  including  bright  sunshine,  dust,  fog,  rain,  snow,  and  darkness  (self  lighting).  And  it  is  not 
just  seeing  through  precipitation  that  can  be  problematic;  bright  sunlight  creates  glints  and  glare,  and  rainwater  and  snow  on 
the  road  or  on  obstacles  can  create  reflective  surfaces  that  make  sensing  difficult. 
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3.  TECHNOLOGY/SENSORS 

There  are  a  variety  of  sensors  that  a  UGV  can  use  to  provide  information  about  its  environment,  including  active  systems 
such  as  RADAR,  LIDAR  and  structured  light,  and  passive  camera  systems,  including  stereo  pairs  and  infrared  sensors. 
Most  of  these  can  be  used  to  provide  a  3D  model  of  the  environment  with  different  levels  of  resolution  and  accuracy. 
Camera  sensors  can  also  provide  color  and  texture  information.  Other  sensors  can  provide  information  about  the  vehicle’s 
state,  including  Inertial  Measurement  Units  (IMU),  odometry  and  velocity  sensors,  to  provide  local  position  information, 
GPS  to  provide  global  position,  various  engine/motor  measurements,  steering  angle  sensors,  and  the  sequence  of  steering 
and  throttle  commands.  The  system  can  also  use  a  variety  of  databases,  including  topographical  and  road  maps,  sensor  data 
from  prior  runs  through  the  environment,  and  rides  of  the  road. 

The  UGV  fuses  data  from  these  various  sources  to  create  an  internal  model  of  its  surroundings.  This  internal  model 
represents  everything  the  UGV  knows  about  the  world  around  it.  The  UGV  uses  this  knowledge  to  plan  its  immediate 
actions  given  the  higher-level  mission  planning  and  goals.  The  quality  of  the  UGV’s  model  depends  on  both  the 
performance  and  complement  of  sensors  available  and  the  UGV’s  ability  to  interpret  the  sensor  data  with  respect  to  its 
mission  and  capabilities.  The  intelligence  portion  of  the  problem  can  become  a  real  bottleneck  in  the  system. 

For  example,  the  sensors  may  be  of  high  enough  resolution  and  dynamic  range  to  allow  humans  to  drive  the  vehicles  in  a 
reasonable  fashion  while  viewing  sensor  feeds.  Although  communication  latency  and  bandwidth  restrictions  can  make 
teleoperation  difficult,  especially  when  accurate  perception  and  quick  decisions  are  necessary  in  complex  high-speed 
operations,  these  are  not  insurmountable  problems.  However,  the  human  operator,  even  with  imperfect  sensors,  is  able  to 
draw  upon  years  of  learning  and  past  experience  to  interpret  the  surroundings  and  to  operate  the  vehicle  appropriately  and 
safely  to  achieve  the  mission  objectives. 

For  the  UGV,  the  most  pressing  technical  problem  remains  the  automated  extraction  and  intelligent  interpretation  of 
relevant,  and  often  subtle,  information  from  multiple  sources  of  data.  Additionally,  the  UGV  must  manage  the  data  in  an 
internal  model  with  sufficient  detail  and  vocabulary  to  allow  intelligent  and  safe  behavior  in  complex  environments. 

The  SOURCE  test  program  and  large  autonomous  UGV  system  provides  a  valuable  platform  to  advance  the  technology  and 
methodology  necessary  to  expand  the  use  of  UGV’s  to  meet  Army  missions.  SOURCE  UGV  used  for  the  Enhanced 
Experiment  included  a  capable  vehicle,  a  suite  of  multiple  sensor  types  and  an  advanced  SOURCE  Autonomy  System  that 
leveraged  technology  from  multiple  DOD  programs.  The  system  is  shown  in  Figure  2.  The  TARDEC  Advanced  Platform 
Demonstrator  (APD)  is  the  primary  UGV  vehicle  used  in  the  experiment.  Additionally,  the  experiment  included  a 
commercial  Jeep-based  vehicle  with  the  SOURCE  Autonomy  System  installed. 
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Figure  2.  SOURCE  UGV  Is  Based  on  the  APD  Vehicle  and  SOURCE  Autonomy  System 
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The  SOURCE  Autonomy  System  is  based  on  technology  leveraged  from  the  Autonomous  Navigation  System  program  and 
advanced  under  the  SOURCE  ATO  to  address  MOUT  mission  requirements.  The  sensor  suite  of  the  autonomy  system 
included  a  high  resolution  scanning  LADAR,  stereo  vision  Electro  Optic  Infra  Red  (EOIR)  imagers,  a  precision  GPS/INS 
navigation  unit  and  COTS  computing  hardware.  The  sensor  suite  provides  data  used  for  the  UGV  to  build  its  model  of  the 
surroundings.  A  key  feature  for  robust  UGV  operation  in  a  broad  range  of  environments  is  the  use  of  a  range  of  sensors  that 
depend  on  different  physical  modalities  for  detection,  including  LADAR  using  laser  ranging,  visible  light  imagers,  Long 
Wave  Infra  Red  (LWIR)  imagers,  Stereo  Vision  processing  and  RADAR  (future). 

In  a  complex  environment,  given  different  atmospheric  and  environmental  conditions,  different  lighting  conditions  and  giv¬ 
en  objects  consisting  of  an  extremely  wide  range  of  materials,  both  natural  and  manmade,  it  is  important  to  have  diverse  set 
of  sensing  technologies.  No  single  sensor  technology  works  well  in  all  situations.  Figure  3  shows  the  types  of  sensor  data 
available  from  the  SOURCE  sensor  suite.  Data  from  all  of  the  sensors  is  used  to  develop  the  SOURCE  UGV’s  model  of  its 
surroundings.  This  model  is  used  to  plan  it  actions  to  achieve  the  mission  objectives.  Figure  4  shows  an  example  of  part  of 
the  internal  model  developed  by  the  SOURCE  UGV. 


Stereo  Disparity 


Figure  3.  SOURCE  Data  from  Multiple  Sensors  Is  Used  To  Interpret  Its  Surroundings 


Figure  4.  Internal  Model  Developed  From  Sensor  Data 


4.  OTHER  PROJECTS  IN  AUTONOMOUS  URBAN  NAVIGATION 

There  are  a  variety  of  civilian  efforts  to  provide  autonomous  operation  of  vehicles  in  urban  environments.  The  Google  Self- 
Driving  Car’s  main  sensor  is  a  LIDAR  that  is  used  to  build  a  dense  3D  model  of  the  environment.  RADAR  is  used  for 
obstacle  detection  and  avoidance  and  a  camera  is  used  to  detect  stoplights.  The  system  is  heavily  dependent  on  a  very 
detailed  map  of  its  route,  presumably  locating  the  position  and  content  of  relevant  traffic  signs  and  potential  static  obstacles. 
The  system  also  depends  on  a  database  of  sensor  data  that  is  collected  during  multiple  runs  along  the  routes  that  it  will  take. 
The  Google  Self-Driving  Car  has  driven  many  thousands  of  miles  in  autonomous  mode  through  urban  environments,  such 
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as  on  the  roads,  streets  and  highways  within  and  adjacent  to  the  city  of  San  Francisco.  However,  a  human  is  always  in  the 
driver  seat,  in  case  problems  occur. 

The  VisLab  group  from  the  University  of  Parma  ran  the  VisLab  Intercontinental  Autonomous  Challenge  (VIAC)  project 
that,  in  2010,  ran  autonomous  vehicles  from  Parma,  Italy  to  Shanghai,  China,  almost  16,000  Km.  The  system  had  a  lead 
vehicle  that  operated  autonomously  much  of  the  time,  but  often  relied  on  humans  to  perform  high  level  path  planning,  since 
no  maps  were  available  for  a  significant  portion  of  the  route.  The  follow  vehicle,  driving  autonomously,  followed  either  by 
detecting  the  lead  vehicle  or  by  following  GPS  waypoints  sent  back  from  the  lead  vehicle.  The  sensor  suite  contained  four 
laser  scanners  and  seven  cameras,  as  well  as  GPS  and  IMU.  The  follow  vehicle  used  local  sensing  to  refine  its  position 
relative  to  the  GPS  waypoints  provided  by  the  lead  vehicle.  When  the  lead  vehicle  was  not  in  view  and  local  sensing  failed, 
the  follow  vehicle  would  have  to  stop  driving  autonomously. 

5.  TESTING 

While  the  various  demonstrations  of  autonomous  vehicle  operation  in  urban  environments  are  impressive,  there  are  still 
shortcomings  that  have  not  been  addressed,  including  driving  completely  autonomously,  without  a  human  in  the  driver  seat, 
through  areas  that  have  not  be  driven  before  or  where  there  have  been  substantial  modifications  to  the  environment.  It  is 
unlikely  that  the  Army  will  be  satisfied  with  these  types  of  demonstrations  alone  when  making  decisions  on  ground  vehicle 
safety.  Given  the  complexity  of  the  real  world  it  is  not  clear  how  many  miles  of  autonomous  driving  would  be  required 
before  the  Army  testing  community  would  deem  an  autonomous  vehicle  safe  to  use.  The  purpose  of  this  paper  is  to  explore 
test  methodologies  that  will  enable  the  Army  test  community  to  have  confidence  that  autonomous  vehicles  can  operate 
safely. 

A  key  safety  metric  for  a  deployable  system  is  the  Lifetime  Hazard  of  the  system.  This  generally  takes  the  form  of  the 
probability  of  a  Critical  or  Catastrophic  Incident  occurring  over  the  expected  life  of  a  system.  For  UGVs,  as  with  other 
complex  systems,  this  may  be  determined  based  on  evaluation  of  numerous  constituent  probabilities/frequencies  of  the 
system  and  its  intended  use.  This  includes: 

UGV  System 

o  Model  performance  probabilities  at  the  major  functional  component  level  (hardware  and  software) 

o  Model  the  overall  system  by  the  combining  of  the  constituent  model  analyses  based  on  the  system  design 
and  operation 

o  Validate  models  against  real  world  testing  at  both  the  component  and  system  levels 

Mission  and  Environment 

o  Model  or  define  aspects  of  the  mission  that  affect  hazards 

o  Account  for  region,  environment,  population,  day/night  operation,  etc. 

o  Include  system  operation  during  non-mission  events,  including  training,  support,  transport,  staging,  etc.  in 
the  analysis 

The  Lifetime  Hazard  depends  on  multiple  aspects  of  the  system  and  the  related  constituent  probabilities/frequencies.  The 
following  notional  list  represents  this  type  of  analysis  focused  on  hazards  associated  with  encountering  pedestrian(s). 

Hazard  Analysis  Depends  On  The  Probability/Frequency  Of: 

1 .  Encountering  a  pedestrian/human 

o  Depends  on  Mission  Type,  Area  of  Operation,  Population,  etc. 

2.  UGV  Detecting  the  pedestrian/human 

o  Depends  on  Surroundings  and  Autonomy  System  Performance 

3.  UGV  Planning  around  the  pedestrian/human 

o  Depends  on  Autonomy  System  Performance 

4.  UGV  Successfully  avoiding  the  pedestrian/human 

o  Depends  on  Autonomy  System  Performance,  Vehicle  Response 

5.  Human  interaction  (e.g.  supervised  autonomy) 

o  Depends  on  Level  of  Autonomy,  Mission,  Area  of  Operation 

6.  Pedestrian  avoiding  collision 

o  Depends  on  assumptions  regarding  human  behavior  and  vehicle  warnings/alerts 
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7.  Collision  with  Pedestrian/human 

o  Depends  on  the  above  components 

8.  Collision  resulting  in  a  “critical”  or  “catastrophic”  incident 

o  Depends  on  Mission,  Vehicle  size/shape,  Autonomy  System 

9.  Lifetime  Hazard 

o  Depends  on  Missions,  Fleet  size,  Operating  Life,  etc. 

The  UGV  system  Lifetime  Hazard  metric  varies  with  vehicle  size/speed,  environment,  mission  type,  region  of  operation, 
population,  fleet  size,  etc.  The  same  autonomy  system  will  have  different  Lifetime  Hazard  ratings  given  different 
applications,  missions,  environments,  modes  of  operation,  etc.  The  Lifetime  Hazard  analysis  also  depends  on  the 
distribution  of  uses  over  the  system  lifetime  (e.g.  operational  mission  time,  support  time,  training  time,  etc.).  The  hazard 
metric  also  depends  on  presumptions  about  pedestrian  behavior.  Pedestrian  behavior  may  be  affected  by  system  design, 
including  warning  indicators  or  horns.  Soldier  pedestrian  behavior  is  heavily  influenced  by  training  and  Tactics,  Techniques 
and  Procedures  (TTP’s). 

As  with  the  evaluation  of  any  complex,  probabilistic  system,  care  must  be  taken  in  defining  and  executing  quantitative  tests 
to  avoid  introducing  unintended  biases  into  the  results.  In  order  for  test  metrics  to  be  generalized,  they  must  be  collected  in  a 
way  that  is  independent  of  the  individual  test  setup,  test  location  or  scenarios. 

In  practice,  this  hazard  analysis  can  become  very  challenging  due  to  the  complexity  of  the  system  and  its  mission.  The 
dimensionality  of  the  problem  space  is  so  large  that  exhaustive,  end-to-end,  testing  is  computationally  infeasible.  However, 
the  described  component  approach  can  decouple  interactions,  thereby  significantly  reducing  the  dimensionality  of  the 
testing  required.  Additionally,  one  could  supply  a  variety  of  inputs  to  the  software  components  through,  for  example,  a 
modeling  and  simulation  environment,  and  then  analyze  the  output  space.  The  use  of  a  modeling  and  simulation  approach, 
integrated  with  real  world  testing  helps  development  and  testing  costs  while  still  retaining  reliability  of  results. 

A  further  complication  is  that  the  intelligence  algorithms  may  include  a  learning  component,  by  which  the  output  may 
depend  on  the  order  that  the  inputs  are  presented  to  the  system.  This  would  lead  to  a  combinatoric  explosion  in  possible 
inputs,  making  the  analysis  nearly  impossible.  The  technology  to  integrate  limited  learning  algorithms  in  UGV’s  is 
emerging  in  research  work.  However,  the  implications  on  formal  system  safety  qualification  of  such  systems  must  be 
addressed  by  the  Army  and  the  technical  community.  What  additional  qualification  methods  will  need  to  be  developed  and 
employed  to  account  for  adaptable  behavior? 

One  could  also  try  to  perform  a  vehicle  driving  “Turing  Test,”  where  one  has  human  judges  trying  to  determine  whether  the 
vehicle  is  driven  by  a  human  or  a  computer.  Such  a  test  may  lead  to  open  ended  questions  and  a  subjective  assessment.  The 
question  then  becomes,  is  it  sufficient  for  an  unmanned  vehicle  to  perform  as  well  as  a  human,  or  will  it  have  to  perform 
better?  Do  we  even  want  the  system  to  perform  like  a  human,  including  all  the  mistakes?  Comparison  with  human 
performance  can  be  problematic  in  and  of  itself  because  of  the  wide  variation  in  human  performance.  Additionally, 
mimicking  human  behavior  along  with  achieving  mission  performance  goals  adds  an  additional  layer  of  requirements  on  the 
system.  However,  these  types  of  comparisons  are  certainly  important  to  make.  These  are  likely  to  be  instructive  for  UGV 
requirements  development  and  also  for  guiding  the  direction  of  algorithm  development. 

Performing  such  a  comparison  leads  to  a  variety  of  challenges.  How  does  one  structure  a  test  environment  to  achieve 
reproducible  results?  How  does  one  measure  “how  close”  the  UGV  came  to  human  performance  and  “how  close”  is  “close 
enough”?  How  does  one  determine  what  is  sufficient  variation  in  urban  environments  and  situations,  such  that  the  results 
can  be  successfully  interpolated?  Very  different  specific  behaviors  may  be  equally  acceptable  in  a  given  situation.  How  does 
one  provide  the  unmanned  vehicle  with  the  experience  that  a  human  driver  would  have?  The  answer  to  the  latter  question 
may  lie  in  modeling  and  simulation,  at  least  for  the  decision-making  algorithms.  Modeling  and  simulation  are  not 
sufficiently  advanced  for  realistic  simulation  of  electro-optical  sensors,  human  perception,  and  pattern  recognition 
capabilities.  These  questions,  among  many  others,  are  a  mere  sample  of  the  vast  concerns  and  issues  that  reside  on  testing 
autonomous  systems.  The  latter  section  will  present  the  solution  taken  by  the  Army  for  the  SOURCE  ATO. 

6.  RESULTS  OF  CURRENT  METHODOLOGY  EXPERIMENTED 

In  planning  for  the  latest  SOURCE  ATO  experiment,  much  debate  occurred  between  development  and  test  engineers  as  to 
the  right  methodology  for  testing  the  safe  operation  of  autonomous  vehicles  in  an  urban  environment.  Although  complete 
agreement  on  what  was  the  best  procedure  was  not  reached,  the  Army  developed  a  preliminary  solution  with  which  to  test 
the  performance  and  behaviors  of  the  SOURCE  ATO  autonomous  vehicle. 

When  testing  a  large  developmental  UGV  in  an  urban  environment,  it  would  initially  be  very  unsafe  to  use  live  humans  in 
close  proximity  to  the  vehicle,  until  greater  confidence  has  been  gained  in  the  ability  to  detect  and  avoid  humans.  Since  the 
UGV  is  the  item  under  test,  the  behavior  of  the  system  cannot  be  presumed  with  sufficient  safety  ahead  of  time.  Therefore, 
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mannequins  were  used  to  represent  personnel  throughout  the  urban  environment.  This  enabled  testing  of  the  system’s  ability 
to  perform  person  detection  and  avoidance,  without  the  danger  involved  if  the  system  did  not  behave  as  expected.  The  team 
built  a  cart-pulley-winch  system  that  would  slide  or  drag  the  mannequins  in  order  to  represent  dynamic  human  motion.  This 
solution  was  reasonably  effective,  although,  initially,  there  were  some  issues  with  test  repeatability.  To  alleviate  this 
problem,  the  mannequin  systems  were  outfitted  with  a  cart  or  a  sled,  depending  on  the  terrain,  and  a  pressure  sensitive 
trigger  device  to  activate  the  mannequin  movement  when  the  autonomous  test  vehicle  approached  a  specific  location.  Even 
with  the  enhancements,  though,  it  was  still  very  difficult  to  confirm  the  behavior  of  the  vehicle  and  collect  the  necessary 
data  to  measure  the  performance  of  the  system.  The  SOURCE  team  implemented  a  very  complex  and  high  fidelity  network 
across  the  urban  environment  to  support  high-bandwidth  data  collection.  Each  mannequin  contained  a  GPS  device  that 
broadcast  its  position,  trajectory,  and  speed,  during  each  test  run. 


Figure  5.  Mannequin  System  Used  To  Emulate  Moving  Pedestrians 


A  great  deal  of  post  processing  is  then  required  to  align  the  position  of  the  autonomous  system  with  respect  to  each  of  the 
obstacles  and  then  extract  information  concerning  the  vehicle  system’s  performance.  Even  though  the  unmanned  vehicle 
may  have  avoided  an  impact  with  an  obstacle,  one  needs  to  analyze  the  data  to  determine  if  the  vehicle  properly  detected  the 
obstacle  and  if  the  trajectory  to  avoid  the  obstacle  was  the  optimal  one.  The  team  structured  the  test  environment  to  include 
as  many  mannequins-to-unmanned  vehicle  interactions  as  practical  given  the  program  logistics.  These  included  situations 
where  the  mannequin  was  static  in  the  road,  at  the  edge  of  the  road,  on  the  sidewalk,  far  from  the  edge  of  the  road,  in 
clusters  or  groups,  moving  at  various  speeds  parallel  to  the  vehicle’s  path,  both  with  and  against  its  path  of  travel,  laterally 
from  either  direction,  and  at  various  angles.  The  vehicle  was  expected  to  ignore  mannequins  whose  trajectory  would  not 
bring  it  sufficiently  close  to  the  platform.  One  limitation  of  the  mannequin  winch  system  is  the  inability  to  conduct  non¬ 
linear  motions.  Although  the  team  did  not  devise  a  system  for  producing  non-linear  dynamic  human  obstacles,  they 
identified  potential  solutions  that  included  robotic  mannequins  to  produce  random  behavior-based  movements  and/or 
waypoint  following  motions. 


Similar  to  the  safety  concern  for  human  obstacles,  the  team  needed  to  develop  a  solution  for  simulating  traffic  within  the 
urban  environment.  This  was  accomplished  by  utilizing  other  robotic  vehicles  that  were  operated  either  via  teleoperation  or 
had  GPS  waypoint  following  capability,  but  without  an  obstacle  detection  and  obstacle  avoidance  (ODOA)  system.  While 
dynamic  vehicles  with  random  behaviors  were  not  implemented,  the  team  thought  it  would  be  interesting  to  utilize  vehicle- 
based  obstacles  with  ODOA  in  order  to  observe  the  interactions  between  two  systems  simultaneously  attempting  to  avoid 
one  another.  The  system  would  need  to  be  fairly  advanced  and  well  tested  before  conducting  such  an  experiment.  It  was 
thought  to  be  too  costly  to  expose  the  current  system  to  that  level  of  testing  at  its  current  stage  of  development. 


In  the  course  of  the  SOURCE  Enhanced  Experiment,  the  SOURCE  system  completed  109  different  runs  through  the  Camp 
Lejeune  MOUT  site.  In  addition  to  the  core  experiment  runs,  the  team  executed  numerous  excursions  or  special  test  runs  to 
gather  additional  data  to  support  future  development  as  well  as  supporting  other  analysis  described  in  this  paper.  The 
excursion  runs  included  runs  at  increased  speeds,  runs  with  alternate  perception  algorithms  enabled,  and  runs  with  a  human 
driver  tele-operating  the  vehicle.  The  runs  are  summarized  in  Table  1. 
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Table  1.  Summary  of  test  runs  during  the  SOURCE  Enhanced  Experiment 


Test  Run  /  Date 

Nominal  Distance 

05  Nov  11 

06  Nov  11 

07  Nov  11 

08  Nov  11 

Total 

Complete  Course  - 
Movers  Off 

1 .5  km  per  run 

3 

2 

2 

2 

9 

Complete  Course 

1 .5  km  per  run 

10 

1 

- 

15* 

26 

Blue  Course 

0.5  km  per  run 

- 

11 

5 

- 

16 

Red  Course 

0.4  km  per  run 

- 

11 

5,  5* 

- 

21 

Black  Course 

0.6  km  per  run 

- 

11 

5,  5* 

6# 

27 

Raw  Data  Collection 

1 .5  km  per  run 

- 

2 

1 

- 

3 

Other 

1 .5  km  per  run 

- 

- 

2°,  3& 

2® 

7 

Total 

98.2  km 

13 

38 

33 

25 

109 

*  Executed  run  at  increased  speed,  except  in  safe  zone  closest  to  base  camp 
°  Tele-op  runs  of  entire  course 

&  Vehicle  maneuvering  runs  at  higher  speeds  with  mover  detection  disabled 

#  Different  sensor  algorithms  turned  on  or  off 
@  MapReg  used  for  entire  test  run 

The  SOURCE  Autonomy  System,  including  data  logging  and  test  tools,  provides  significant  capability  to  monitor,  store  and 
later  analyze  a  large  amount  of  data  in  real  time.  As  described  above,  this  provides  the  critical  capability  to  review  and 
replay  data  from  a  test  run.  The  data  can  be  analyzed  and  post  processed  in  a  wide  variety  of  ways.  It  provides  visibility  into 
the  real  time  data  that  result  in  the  decisions  the  Autonomy  System  makes.  It  is  generally  not  possible  to  deduce  this 
information  by  only  observing  the  UGV  operation.  This  capability  is  by  no  means  unique  to  SOURCE,  but  it  is  key  to 
advancing  the  development  and  assessment  methodologies  for  UGV’s  in  more  complex  environments. 

The  data  logged  includes  much  of  the  internal  data  transmitted  between  major  functional  elements  within  the  SOURCE 
Autonomy  Software.  This  data,  combined  with  ground  truth  data  collected  as  part  of  the  experiment  runs,  can  be  used  to 
measure  key  performance  values  important  for  verifying  system  operation.  For  example,  Figure  illustrates  several 
parameters  assessed  for  a  given  pedestrian  encounter  on  one  specific  run.  This  data  is  from  the  encounter  with  Moving 
Mannequin  1  during  Run  24  on  6  November.  The  figure  shows  the  range  at  which  the  pedestrian  was  initially  detected,  as 
well  as  the  amount  the  UGV  deviated  from  an  ideal  path  to  avoid  the  moving  pedestrian. 
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Figure  6.  Example  of  Quantitative  Results  Based  On  Logged  UGV  Data  With  Ground  Truth  Data 

These  are  only  two  examples  of  the  type  of  quantitative  assessment  that  may  be  performed  if  the  right  data  is  logged  and 
maintained.  The  analysis  of  the  data  can  also  be  used  to  investigate  problems,  to  initiate  design  improvements  or  even  to 
perform  more  advanced  analyses  that  may  not  have  been  considered  at  the  time  of  the  actual  test. 

In  addition  to  the  quantitative,  digitally  logged  data  for  each  run,  as  described  above,  the  SOURCE  Enhanced  Experiment 
test  methodology  included  recording  detailed  notes  for  each  run  executed,  including  the  number  of  times  the  vehicle 
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departed  the  travel  lane  and  went  over  the  curb,  each  time  the  system  paused,  and  every  time  the  safety  driver  in  the  chase 
vehicle  had  to  take  control  due  to  unsafe  maneuvering  of  the  APD.  These  notes  were  compiled  from  test  personnel 
witnessing  the  runs  from  the  test  control  center,  as  well  as  from  personnel  in  the  chase  vehicle  following  the  UGV  on  all 
runs. 

In  an  attempt  to  provide  evidence  for  the  performance  of  the  autonomous  vehicle  versus  the  performance  of  human  drivers, 
the  SOURCE  team  conducted  a  parallel  test  that  used  human  operators  with  good  driving  records  teleoperating  the  vehicle 
through  the  test  course  under  the  same  conditions  as  the  autonomous  vehicle.  The  operators  were  not  briefed  on  the  route 
ahead  of  time  and  were  given  directions  verbally  throughout  the  test  scenario.  The  operators  would  notify  the  data  recorders 
when  they  saw  and  when  they  recognized  an  obstacle,  and  they  were  encouraged  to  think  out  loud  throughout  the  test.  The 
vehicle  was  equipped  with  advanced  data  collection  software  that  recorded  the  vehicle’s  location,  trajectory,  and  speed,  as 
well  as  recorded  the  GPS  locations  when  the  operator  detected  and  recognized  obstacles.  In  later  analyses,  this  enabled 
engineers  to  do  a  side-by-side  comparison  of  the  autonomous  behaviors  and  the  human  operated  behaviors. 

While  maintaining  the  established  speed  limits  in  the  urban  environment,  limits  that  ranged  from  10  to  25  miles  per  hour, 
the  human  operators  on  average  completed  the  course  2.5  to  3  times  as  fast  as  the  autonomous  vehicle.  These  results 
highlight  key  requirements  tradeoffs  that  often  arise  with  new  technology  in  complex  missions.  In  an  operational  UGV 
system,  safety  performance  at  an  individual  pedestrian  encounter  level  is  at  odds  with  mission  effectiveness,  especially 
speed.  This  is  similar  to  many  other  decision-based  systems.  As  autonomous  UGV  technologies  emerge,  the  early  systems 
are  designed  to  employ  a  “when  in  doubt,  slow  down”  approach.  This  allows  technical  development  to  proceed  while 
maintaining  safety. 

In  the  long  term,  when  faced  with  uncertainty,  should  the  UGV  err  on  the  side  of  minimizing  risk  or  maximizing  perfor¬ 
mance?  Human  operators  will  often  continue  on  their  current  path  of  travel  even  if,  for  example,  the  sun  glares  in  their  face, 
or  something  else  interferes  with  their  ability  to  perceive  the  environment  clearly.  The  SOURCE  autonomous  system  is  cur¬ 
rently  programmed  to  err  on  the  side  of  safety  rather  than  speed.  Again,  in  the  long  term,  this  Risk  vs.  Performance  tradeoff 
may  need  to  be  based  on  operator  direction  to  the  UGV,  analogous  to  adjustable  “rules  of  engagement”.  This  allows  the 
operator  to  balance  UGV  operation  based  on  the  known  situation.  Of  course,  this  increases  system  complexity  as  well  as 
difficulty  in  system  qualification.  Either  the  level  of  adaptability  or  the  specific  performance/risk  trades  need  to  be  explicit 
in  future  system  requirements. 

7.  CONCLUSION  AND  FUTURE  RESEARCH  CONSIDERATIONS 

Regardless  of  how  much  data  is  collected  and  how  the  system  performs  in  100,  1,000,  or  even  10,000  experiments,  the 
question  still  arises  as  to  the  methodology  necessary  to  reliably  measure  or  assign  a  safety  level  for  the  system  and  expect  it 
to  be  accurate  for  a  reasonably  broad  set  of  environments  and  conditions.  What  metrics  are  used  and  at  what  performance 
level  is  the  vehicle  approved  as  safe?  Humans  operate  vehicles  every  day,  having  different  ages  and  educational 
backgrounds,  and  often  in  very  complex  hazardous  driving  situations  and  conditions,  yet  the  general  public  are  comfortable 
with  that  level  of  risk.  When  you  then  list  the  same  conditions  and  complex  situations,  but  use  an  autonomously  navigated 
and  operated  vehicle,  the  public  becomes  extremely  hesitant  and  concerned  with  the  safety  of  the  system  and  its 
unpredictability.  A  mature  autonomous  vehicle  is  likely  to  be  far  more  predictable  then  the  average  human  driver. 
Autonomous  vehicles  do  not  have  stressful  work  deadlines  or  personal  issues  rolling  around  in  the  back  of  their  head  while 
they  are  driving,  and  do  not  get  distracted  while  talking  on  the  telephone,  singing  along  to  music,  or  eating,  while  they  are 
driving,  yet  still  they  are  feared  by  the  public. 

Continued  evolution  of  the  Lifetime  Hazard  analysis  presented  in  this  paper  will  be  valuable  to  achieving  the  necessary 
safety  performance  of  UGVs  in  the  future.  This  work  can  form  the  initial  framework  but  it  depends  on  inputs  form  a  wide 
variety  of  key  stakeholders  moving  forward,  including  the  safety  community,  test  and  verification  community,  user 
community  and  development  community.  This  framework  augments  the  existing,  well  established,  safety  processes  and 
methods  by  focusing  on  some  of  the  unique  challenges  of  complex,  autonomous  UGV  systems  and  missions.  It  is  important 
to  note  that  key  parameters  in  the  framework  depend  on  assumptions  about  mission,  human-UGV  encounters,  reaction  of 
humans  and  other  characteristics  outside  the  UGV  system  itself. 

There  is  room  for  advancement  in  the  area  of  UGV  requirements  definition.  Clear  definition  of  the  intended  mission  types 
and  expected  use  can  help  bound  the  complexity  and  cost  of  any  particular  system.  UGV’s  may  be  categorized  by  their 
Level  of  Autonomy.  The  exact  definitions  vary  somewhat  but  conceptually  this  indicates  the  level  of  decisions  the  UGV 
may  make  without  help.  At  higher  Levels  of  Autonomy,  the  UGV  is  expected  to  handle  a  broader  set  of  conditions  without 
human  assistance.  At  lower  Levels  of  Autonomy,  the  UGV  operation  depends  on  assistance,  oversight  or  direct  control  by  a 
human.  Availability  and  response  time  of  a  human  to  assist  or  human  safety  operator  to  override  affects  both  safety  and 
mission  effectiveness.  The  same  UGV  system  can  have  a  much  higher  level  of  safety  if  it  is  operated  at  lower  level  of 
autonomy. 
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As  described  in  this  paper,  the  test  methodology  for  developing  and  assessing  autonomous  UGVs  has  multiple  facets  to 
address  the  broad  nature  of  the  UGV  stakeholders.  Continued  development  of  the  detailed  test  methodology  is  necessary  to 
best  balance  the  needs  of  all  stakeholders.  Again,  much  of  this  builds  on  established  test  approaches  employed  on  other 
complex  systems.  Detailed  quantitative  testing  at  the  level  of  key  intelligent  components  is  necessary.  This  testing  should 
focus  on  simple,  repeatable,  high  repetition  tests.  The  specificity  and  quantity  of  data  must  support  determining  the  key 
probabilities  such  as  those  defined  in  the  Lifetime  Hazard  discussion  above.  The  fidelity  of  the  quantitative  results  must  also 
support  validating  any  probabilistic  models  used  in  the  hazard  analysis.  These  are  closely  coupled  to  the  UGV  system. 

End-to-end  test  scenarios  are  also  necessary.  This  focuses  on  more  general  tests  and  is  closely  coupled  to  the  required 
capabilities,  intended  mission  and  environment.  These  include  lower  repetition  rates,  but  broader,  more  operationally 
oriented  tests. 

Development  of  testing  methodologies  for  comparison  of  UGV  operation  to  human  drivers  is  needed.  There  is  a  large 
number  of  approaches  to  and  directions  that  this  type  of  comparison  can  take  and  there  is  still  a  lot  of  research  and 
investigation  needed  to  evolve  the  methodology  in  this  area. 

This  paper  described  numerous  challenges  to  Autonomous  Navigation  of  UGV’s  as  the  intended  use  and  target  environment 
becomes  more  complex.  The  SOURCE  Enhanced  Experiment  at  Camp  Lejeune,  NC  in  2011  was  used  as  a  platform  to 
exercise  some  key  aspects  of  the  development  and  methodology  that  the  Army  has  sought  to  increase  UGV  mission 
capabilities.  The  SOURCE  program  will  conclude  with  the  Capstone  Experiment,  also  at  Camp  Lejeune  in  September  and 
October  2012.  The  accomplishments  presented  represent  some  initial  progress  in  the  direction  necessary  to  help  realize  the 
goal  of  using  autonomous  systems  to  exchange  manned  ground  vehicles  for  UGVs.  Future  work,  beyond  the  SOURCE 
program,  building  on  the  progress  presented  in  this  paper,  is  necessary  in  both  UGV  technology  development,  as  well  as 
UGV  testing  and  hazard  assessment  methodology.  This  work  will  help  ensure  that  UGVs  will  effectively  meet  the  Army’s 
objectives. 

**Disclaimer:  Reference  herein  to  any  specific  commercial  company,  product,  process,  or  service  by  trade  name, 
trademark,  manufacturer,  or  othei'W’ise,  does  not  necessarily  constitute  or  imply  endorsement,  recommendation,  or  favoring 
by  the  United  States  Government  or  the  Department  of  the  Army  (DoA).  The  opinions  of  the  authors  expressed  herein  do  not 
necessarily  state  or  reflect  those  of  the  United  States  Government  or  the  DoA  and  shall  not  be  user  for  advertising  or 
product  endorsement  purposes.  ** 
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